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5 SYSTEM AND METHOD TO CONTROL NETWORK DEVICES ON A PALM SIZED COMPUTER 

LIMITED COPYRIGHT WAIVER 

10 

A portion of the disclosure of this patent document contains material to which the claim 
of copyright protection is made. The copyright owner has no objection to the facsimile 
reproduction by any person of the patent document or the patent disclosure, as it appears in the 
official file or records, but reserves all other rights whatsoever. 

BACKGROUND OF THE INVENTION 

1 5 Field of the Invention 

This invention relates to the field of networked computer. In particular, the invention 
relates to a system and method to control network devices using a palm sized, or otherwise 
reduced functionality, computer. 

20 Description of the Related Art 

Palm sized computers, also referred to as Personal Digital Assistants (PDAs), are 
portable devices which perform an array of personal management tasks such as calendar 
management and address book storage. The adoption of palm sized computers has been rapid. 
Some palm sized computers are able to interface with conventional computing devices, such as 
25 PCs, on an as-needed basis. For example, palm sized computers such as 3Com's Palm 
Platform™ computers can upload personal appointments to a PC-based calendar. 

Palm sized computers generally have the following characteristics. Relative to desktop 
and laptop computers, palm sized computers have limited processing, display and input 
capabilities. As a result of these limitations, palm sized computers do not run the same 
30 applications as desktop or laptop computers. Other limitations of palm sized computer include 
limited batter life and lower bandwidth communications with other devices. 

One big advantage of palm sized computers is their portability. Therefore, it is 
desirable to be able to access desktop functionality from palm sized computers. 
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SUMMARY OF THE INVENTION 

Controlling network services using palm sized computers is described. A program on 
the palm sized computer is used to access a registry of network services that may be available. 
The registry includes descriptions for various services. Each description includes at least a 
reference to program code that can be downloaded to the palm sized computer. Executing this 

10 program causes the palm sized computer to issue commands directly to one or more of the 
specific network services needed. In some cases, these network services include application 
services for running desktop applications that the palm sized computer could not execute. 

In some embodiments, the device executing the network services and the palm sized 
computer are executing middleware application for communicating with the registry. In some 

15 embodiments, this middleware includes Jini technology from Sun Microsystems. Additionally, 
the programs downloaded can include Java program code. 



2 

SUBSTITUTE SHEET (RULE 26) 

RNRDOCID:<WO 0043891 A 1 l > 



WO 00/43891 



PCT/USOO/01634 



5 

BRIEF DESCRIPTION OF THE FIGURES 

Figure 1 illustrates a system having a palm sized computer controlling operations of 
various network devices. 

Figure 2 illustrates a software architecture for use in the system of Figure 1 . 
Figure 3 illustrates an example architecture for controlling a PowerPoint presentation. 
10 Figure 4 illustrates a detailed software architecture for the example of Figure 3. 

Figure 5 illustrates a class hierarchy that can be used in the example of Figure 3. 
Figure 6 illustrates an example class hierarchy for network services that can be 
provided in the system of Figure 1. 

Figure 7 illustrates an example class hierarchy for application services that can be 
15 provided in the system of Figure 1. 

Figure 8 is an architecture diagram for one embodiment of the present invention. 
Figures 9- 1 6 are representative Graphic User Interface constructs for one example of 
the present invention for the control of a slide presentation application. 
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DETAILED DESCRIPTION 

A, System Overview 

A palm sized computer, or other thin platforms having small sizes and displays with 
display areas typically smaller that 4 inches by 6 inches, can serve as a network portal to usher 
in a new generation of mobile computing. Palm sized computers are the ideal next generation 

10 computing device in that they are inherently mobile and have the lightweight form factor 
necessary for mobile computing. The fundamental obstacle to palm sized computers as the 
next generation computing device can be removed by viewing the network as an extension of 
the palm sized computer's resources. Functions can be downloaded into the device as needed, 
and overlaid after they have been used. This allows the palm sized computer to adapt to a 

1 5 changing environment (as the mobile user's location changes) and to access exactly the set of 
services it needs. These services are stored on the network and can be used at will. Many of 
these services may never reside on the device and are more suitable for execution by a 
conventional computer. However, they are accessible and can be controlled via a lightweight 
computing device, such as a palm sized computer. 

20 

B. Building Blocks of Network-enabled Palm Sized Computers 

The building blocks for lightweight mobile computing include a palm sized computer, a 
compute/memory/storage-intensive device(s), and a network. 

Middleware allows palm sized computers to discover network-based computing 
resources. Once discovered, this middleware provides a mechanism for the palm sized 
25 computer to use these resources. This middleware typically includes a directory of resources 
(or services), a protocol for storing and retrieving from the directory, and mechanisms to 
transfer software from the directory to a palm sized computer. 

Control applications reside on a lightweight computing device, such as a palm sized 
computer, but manipulate computing services on the network. Control applications assume the 
30 use of middleware, such as Sun Microsystems Jini, for access to network- 



BNSDOCIO <WO 0043891A1_I_> 



4 

SUBSTITUTE SHEET (RULE 26) 



WO 00/43891 PCT/US00/01634 

5 based resources. (Note in some embodiments, the control application is distributed between the 
palm sized computer and a control proxy server.) 

Example Registry and Control Infrastructure: Jini 

Jini™ is a technology developed by Sun Microsystems which addresses the problem of 
computing and network complexity. It eases the burden of accessing services by providing 

10 seamless access and automatic registration of network resources. Jini accomplishes this by 

adopting a model in which devices announce their presence and capabilities to the network, and 
access the same type of information in order to locate services they wish to use. 

The Jini approach relies on Java and a Jini "registry" (i.e. database of services) as the 
underlying infrastructure. Each device is expected to run a Java Virtual Machine (JVM), or 

1 5 rely on a Jini proxy which runs a JVM on the device's behalf. Key to Java is the idea that 
software (as well as data) can be dynamically downloaded to a device. The Java model 
assumes a distributed, network-centric model in which the behavior of a device can be 
dynamically altered to accommodate changing conditions. 

Jini eases network connectivity problems. Jini acts as middleware to access network 

20 resources, as it lets devices locate services and download software for those services. Other 
middleware could be substituted for Jini if it provides discovery and software download for 
network-based services. 

C. Definitions 

A control device is as a device which runs Jini (or some other discovery and software 
25 download technology) and is capable of accessing a wide range of network-centric services, 
including services which are incapable of running on the control device. The control device is 
the access device for a wide class of computing services otherwise inaccessible to devices with 
restricted computing power. 3Com's palm sized computer, the Palm Computing platform, is an 
example of such a control device. Windows CE compatible devices may also be used. 
30 A control application is as an application that relies on resources located off of the 

control device (e.g. on a network), but uses a control device to initiate and control the 
application. The resources are accessed and controlled, but not resident, on the control device. 
Examples of such compute/memory-intensive services include PowerPoint slide presentations 
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5 and speech recognition systems. These services can be both invoked and controlled via a 
control device. 

Network-based services (rather than devices) can be used by any control device. 
Services offer a discrete task to the control device. This contrasts with a traditional model in 
which devices, such as a PC, house the entirety of resources a control device might utilize. 

10 Services are fine-grained and represent a distributed set of capabilities residing on a network. 
Services may or may not be co-located with other services on the same physical device. 
Services are offered (via a network and a Directory of Services, such as the Jini Lookup) in as 
small a unit as is justifiable given the application tasks users will typically want to accomplish. 
Fine-grained services can be used by a variety of consumers who need precisely that function. 

15 The model that emerges is a network-based model that offers a wide range of narrowly- 
defined computing services of interest to a wide range of applications. Services will be 
physically distributed across devices but accessible via a central repository (e.g. database) of 
services. This model of distributed computing is inherently scalable in that the set of services 
offered via the network grows seamlessly as devices "plug in" and announce their capabilities. 

20 

D. An Example of the Control Application 

To help illustrate the use of control device and a control application, a PowerPoint slide 
presentation control described herein. (Figure 1 illustrates a generalized view of such an 
example.) In this example, a palm sized computer 100 accesses network 110 resources (via a 
Directory of Services, such as the lookup service 120) to locate the elements it requires to 
25 remotely control a presentation located on the network. The palm sized computer 100 uses 
three services to function as a control device to display a PowerPoint presentation: 

_ an application service 130 (to run PowerPoint) 

_ a persistent storage service 150 (to store the presentation) 

_ a display service 140 (to display the presentation images) 
30 Each of these services will have registered with the lookup service 100. A service in 

this example, is tied to different hardware devices. However, this is not required. Storage, 
processing, and/or display could be supported by one or more computer systems. 

None of these services are resident on the palm sized computer 100. Once the palm 
sized computer 100 has located the necessary services, it downloads the code required to 
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control those services (using the lookup and download protocols). Middleware, such as Sun's 
Java/Jini technology, is used to move the code. 

The palm sized computer 100 is then capable of directly controlling the services it 
requires. 

In this example, all the devices can communicate on the network 110, however, they 
need not all be connected to the network in the same way. 
Functionally, the devices play the following roles: 
_ palm sized computer 100: the remote control device 

application services: a set of resources and services, such as compute power and 

applications 

network 110: the physical medium which connects devices and services 
lookup service 120: a database of network services 

E. Architecture 

As noted above, the palm sized computer 100 functions as the remote control device for 
the PowerPoint presentation. It is a multi-function control device in that it can control a host of 
resources accessible via the network. The palm sized computer 100 accomplishes this via 
middleware (e.g. Jini) and a generic control protocol capable of issuing control commands to 
an offboard resource. The palm sized computer 100 implements this control via the software 
components depicted in Figure 2. These software components represent a generic architecture 
for control of any network-based resource using a reduced functionality computer such as a 
palm sized computer. The software architecture, middleware, and control protocol implement 
a new model for lightweight mobile computing. This model of lightweight mobile computing 
is particularly well-served by devices such as a palm sized computer, as they possess the 
requisite size and portability. 

In Figure 2, the control device 200 is able to control services on the network 1 10, such 
as the network based service 250. The directory of services 220 provides the registry functions 
used to enable the system. 

The control device 200 includes a control device operating system 201 which supports a 
network communications program 202 and a control application 210. The network 
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5 communications program 202 allows the control device 200 to communicate with other devices 
on the network. The control application 210 includes the following elements: 

a GUI 212 to display the available network-based services and accept user input 

an application control protocol manager 214 which interfaces between the control device 

200 and the network based computer service 250 by requesting tasks from that service (e.g. 

10 slide manipulation). This module is responsible for generating the application control 

protocol to command the selected service. 

a middleware protocol manager 216 to transfer data between the control device 200 and the 
directory of services 220 (e.g. communication between Palm platform and the Jini 
Lookup). This module is responsible for generating the syntax necessary to communicate 
1 5 based on the type of middleware being used. 

The service device operating system 251 includes a service device communications 
program 252 and a service control application 260. The service device communications 
program 252 is responsible for communicating with the network 1 10. The service control 
application 260 includes the following components: 
20 _ a service application control protocol manager 262 that interfaces between the network 

based computer service 250 and the control device 200 and accepts control tasks (e.g. slide 
manipulation) issued from the control device 200. This module is capable of sending back a 
response (e.g. status) to the control device 200. 

a middleware protocol manager 266 to transfer data between the network based computer 
25 service 250 and the directory service (e.g. communication between the application service 

130 and the Jini Lookup). This module is responsible for generating the syntax used to 
communicate based on the type of middleware being used. 

an application 268 to perform functions on the network. The application 268 can be a 
desktop application that would not execute on a palm sized computer. 
30 Returning to the specific example of the PowerPoint presentation, Figure 3 illustrates 

the architecture that could be used to implement such a system. In Figure 3, the control device 
200 has been replaced with a palm sized computer 100 executing the Palm OS 301. The GUI 
212 is specifically for PowerPoint control (see Figure 1 for an example of such a GUI). The 
Java Virtual Machine 318 is executing on the palm sized computer 100 and replaces the 
35 middleware 21 8. The middleware protocol manager 216 supports Jini discovery, lookup and 
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download protocols. PowerPoint control commands are issued to the network based computer 
service 250. The PowerPoint control protocol manager 364 provides the interface for these 
commands and controls the PowerPoint application 368. 

F. Control Flow 

The process for accessing and controlling network-based services is described below. 
The specific implementation is control of a PowerPoint Service via a 3Com Palm Computing 
platform is used as an example. 

1. Configure a computer hosting the directory of services 220 and connect it to the network 110. 
For instance, a Jini lookup service is configured to listen at a preset TCP, or UDP, port for 
service registration or service lookup requests. 

2. Establish a network connection from the network based computer service 250 to the network 
110. 

30 Register the computing service with a directory of services 220. For example, in the Jini 
model, the computing service initially sends out a multicast packet announcing its presence on 
the network. Once the directory service 220 receives the packet, it sends a unicast packet back 
to the announcer. The packet includes an interface (e.g. Java code) for uploading code back 
to the directory service 220 as well as for searching and downloading code from the directory 
service 220. 

40 Upload service description to the directory service 220. If Jini is adopted as the middleware, 
the application service 130 receives the response from the directory service 220 and uses the 
included interface to upload its service interface to the directory service 220. When the service 
interface is called, it contacts the director}' service 220 which in turn creates an entry 
(represented by an object) for this new service and sets the proper fields such as service name, 
attributes and optionally the corresponding service interface. Other middleware may choose 
to use protocol-based approach such as FTP or TFTP for the uploading process. 

50 Register the storage service 150 and display service 140 via the same process. 

60 Establish a network connection from the control device 200 to the network 1 10. For Palm 
computers, there are multiple options for network connectivity. Possible solutions include 
using the infrared (IR) port to talk to a IR-LAN bridge or router, using the serial port to talk 
to a serial-to-LAN bridge or router, using either the IR or the serial port to talk to a digital cell 
phone and dial up a modem server, and/or using wireless data communications. 
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5 70 Launch the service control graphical user interface (GUI) 212 on the control device 200. 

80 Via the service control GUI 212, accept user input, such as the selection of an application (for * 

example, a PowerPoint application) to be controlled. 
90 Optionally register the control device 200 with the directory service via a registration protocol, 
such as the Jini Discovery Protocol. This step is the same as the above one for the other 
10 services. It is executed only if the control device 200 has resources to offer. 

100 Search the directory service 220 and download the desired service descriptor. In the case 
of Jini, after the control device 200 receives the response from the directory service 220, it 
uses the included interface to search the directory service 220 for an application service using 
the object type representing the service (such as an object type of PowerPoint presentation 
15 service) and the desired service attributes (such as the name and the physical location of the 

service). Once the directory service 220 finds such a service entry, the control device 200 is 
notified, which in turn uses the downloading interface to download the application service 
descriptor. One example of these services is the GUI code for controlling a PowerPoint 
presentation. 

20 110 Send requests from the control device 200 to the network based computer service 250 to 
control the desired application. For example, a "next slide" request could be sent from a palm 
sized computer 100 to an application service 130 running PowerPoint. The communication can 
be based on a protocol such as the following one: 

25 Control Type | Application | File Name J Control Function 

Where: Control Type = {Request, Reply} 

Application = {PowerPoint, FAX, Print, Email, Phonebook } 

File Name = {3ComPalmVIISpecification} 

30 Control function = {File Load, Slide Forward, Slide Backward, File Close } 

Alternatively, techniques such as Java's Remote Method Invocation (RMI) can be used 
to achieve the same goal. In this case, the control device makes a local function call such as 
doForwardSlide(). The RMI mechanism will transfer the call to a remote machine which 
implements and carries out the function call. The PowerPoint presentation service may in turn 
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use other services such as the storage service 150 and the display service 14Q. The procedure 
to employ these services is similar to steps 10-11 above. 

120 Accept any response to requests sent from the control device 200 to the network based 
computer server 250 and process any errors. 

G. Control Device GUI 

An important element of the control application 210 is a GUI front-end which accepts 
user input for controlling the PowerPoint presentation (or other application) and a control 
protocol manager backend which takes user input and translates it into commands to the CPU 
service. An example GUI is depicted in Figure 1 . The example GUI allows the user to click 
on "forward", "backward", "go-to-first-page" or "go-to-last-page" buttons to control the slide 
show. The user can also click a "get-Iist M button to get a fiill list of the slide titles in the current 
presentation and choose to go to a particular slide. By clicking the "scribble" button, the 
window switches to graphics mode. In this mode, the user can draw at random on the panel, 
and the result of the drawing will be sent to the CPU service and eventually displayed on the 
projection service. 

As explained in the software flow section, there are multiple ways to implement the 
application control protocol manager 214, the following illustrates one approach. The protocol 
takes the following form: 

Control Type [ Application | File Name [Control Function 

Where: 

Control Type = {Request, Reply} 

Application = {PowerPoint, FAX, Print, Email, Phonebook, ....} 
File Name = {3ComPalmVIISpecification} 

Control function = {File Load, Slide Forward, Slide Backward, File Close, 

When the user clicks any button or draws something on the GUI, the application 
control protocol manager 214 generates the corresponding field in the protocol and sends a 
command to the network based computer service 250 via a TCP/IP channel. 

11 
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H. Service Control Application Design 

Figure 4 illustrates a detailed software architecture for the service control application of 
Figure 3. The architecture includes three main elements: a CPU service 410 (corresponding to 
the application service 130), a storage service 420 (corresponding to the storage service 150) 
and a projector service 430 (corresponding to the display service 140). Each of the elements 
10 include middleware protocol layer management modules. Each module has a corresponding 
service application control module (e.g., PowerPoint Control Protocol Manager). To control 
specific network services, instances of those services are instantiated and are used by the 
corresponding control protocol manager. The following describes examples of such classes that 
can be instantiated for specific network services. 



15 



PowerPoint Control 



The PowerPoint control, through OLE automation, instantiates and controls an instance 
of PowerPoint application. The PowerPoint control also communicates with the storage service 
420 to store/retrieve presentations and the projector service 430 to view slides. 

The PowerPoint control has the ability to have many presentations open at the same 
20 time and is capable of switching between presentations. A presentation has a collection of 
slides in it. Once open, the PowerPoint control allows easy traversal of the slides either by 
commands like previous, next slide or by direct access (e.g. slide number or slide title). When 
any change in slide position occurs, the PowerPoint application automatically generates the 
image that needs to be shown by the projector control and invokes the method on the projector 
25 control to show the changed slide. 

Once a presentation is open, the PowerPoint control can have facilities such as Add 
Comments and Add Scribbled Graphics to the current slide. Additionally, it can allow the 
adding of new slides to the presentation. 

Figure 5 illustrates an example object class hierarchy for the PowerPoint control class. 
30 The following describes the elements of Figure 5 in more detail. 

PowerPoint 



Property/Method Name 



Type Description 
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Presentations 


Collection 


List of open presentations 


Presentation 


Property/Method Name 


Type 


Description 


Slides 


Collection 


Array of slides in presentation 


Location 


Storage 


Location where presentation loaded from 


View 


Projector 


Projector where presentation is viewed 


Slide 


Property/Method Name 


Type 


Description 


Title 


String 


Title for the slide 


AddComments 


Method 


Add given comments to slide at given co- 
ordinates 


AddScribble 


Method 


Add given graphics to slide at given co- 
ordinates 


Storage 


Property/Method Name 


Type 


Description 


Host 


String 


LDAP Server host where the presentation 
stored 


FileName 


String 


File name and other details for 
presentation 


Projector 


Property/Method Name 


Type 


Description 


Host 


String 


Projector host where slides shown 


ShowSlide 


Method 


Show slide image on projector 
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LDAP Database Control 

The LDAP database control provides file system services to store and retrieve 
presentations. This control gets commands from the PowerPoint control through the LDAP 
protocol, to search for presentations and return presentations. 
10 An object class hierarchy for CLdapDB could include CLdapDb having to a specific 

storage system reference. 

Storage 



Property/Method Name 


Type 


Description 


FileFilter 


String 


File filter used to get file 
list 


FileList 


Collection 


List of files found on 
storage 



15 

Projector Control 

The projector control provides image viewing services and has a simple image viewer 
that shows the image on a projection screen, monitor, display device or canvas. The control 
gets commands from the PowerPoint control. Specifically, to display images, the projector 
20 control could cause frames generated by the PowerPoint control to be displayed at the device 
controlled by the projector control. Other embodiments use more sophisticated techniques for 
displaying the PowerPoint information (e.g., support windowing system API calls that an 
application may make). 

An object class hierarchy for CProjector could include CProjector having to a specific 
25 image view reference. 

ImageView 



RW5?nnr:ir> <wn otm389iai i > 



14 

SUBSTITUTE SHEET (RULE 25) 



WO 00/43891 



PCT/US00/01634 



Property/Method Name 


Type 


Description 


FileName 


String 


rue UdlllC IL> oc ui5>pidycu 


View 


Method 


Show the image on the 
projector 


Mode 


Long 


FullScreen mode 



5 

I. Directory of Services Design 



The directory of services 220 encodes the set of services available on the network 110. 
The director) of services 220 describes the characteristics of these services and provides a 
means to locate those services. To illustrate this concept, an object-oriented directory service 
10 is used. The directory service will have objects whose attributes describe the features of 

available services and optionally include either code to invoke those services or a reference to 
such code. A directory service will typically be one of several components offered in the 
middleware a control device 200 will use. 

Figure 6 illustrates an object class hierarchy which models several network-based 
15 services. These network-based services are Application Service, Operating System Service, 
Storage Service, Projection Service, and Service Location Service. Examples of each of these 
services are now given. 

Define-Class Service 
20 Superclass: root 

Attributes: 
Name 
Status 



25 Define-Class NetworkService 
Superclass: Service 
Attributes: 

Location 

PhysicalAddress 

15 
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5 PhysicalMachine 
Owner 
Vendor 
Version 

1 0 Define-Class ApplicationService 
Superclass : NetworkService 
Attributes: 

CodeLocation 
SerialNumber 
1 5 SupportedFeatures 

Define-Class OperatingSystemService 
Superclass : NetworkService 
Attributes: 
20 RealTimeOS? 

Define-Class StorageService 

Superclass: NetworkService 
Attributes: 
25 DatabaseType 
Schema 

Define-Class ProjectionService 
Superclass : NetworkService 
30 Attributes: 

Resolution 

Define-Class ServiceLocationService 
Superclass : N etworkService 
35 Attributes: 

16 
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Query Protocol 
Schema 

Figure 7 illustrates a partial object class hierarchy which models Application Services, 
including the PowerPoint Application Service: 

Define-Class SlidePresentationService 
Superclass: ApplicationService 



Define-Class GroupwareService 
15 Superclass: ApplicationService 

Define-Class EmailService 

Superclass: ApplicationService 

20 Define-Class SpeechRecognitionService 
Superclass: ApplicationService 

Based on this object class hierarchy, objects which represent network services can be 
defined. These objects are stored in a directory of services 220 1 such as a Jini Lookup. 
25 An object to instantiate a SlidePresentationService could look as follows: 

Make-Instance SlidePresentationService 

Name "PowerPoint" 

Status "Active" 
30 Location "3Com Intranet" 

PhysicalAddress "Building300.Floor2.Cube323" 

PhysicalMachine "PowerBook G3" 

Owner "Elaine Lusher" 

Vendor "Microsoft" 
35 Version "98" 
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CodeLocation "system/applications/office/powerpoint: " 
SerialNumber "169-43-4666" 
SupportedFeatures "Scribble" 

Other examples of objects which represent application services include: 



Make-Instance SpeechRecognitionService 
Name "Naturally Speaking" 
Status "Active* 
Location "3Com Intranet" 
15 Physical Address "Building300.Floor2.Cubel00" 

PhysicalMachine "Solaris" 
Owner "Wenjun Luo" 
Vendor "Dragon Systems" 
Version "4.1" 

20 CodeLocation "system/applications/research/dragonspeech" 

SerialNumber "157-89-4323" 
SupportedFeatures "Dictation for Microsoft Word" 

Make-Instance GroupwareService 
25 Name "Alta Vista Forum" 

Status "Active" 
Location "3Com Intranet" 

PhysicalAddress "Building300.Floor2.Cube220" 
PhysicalMachine "Windows NT" 
30 Owner "Paul Huard" 

Vendor "Microsoft" 
Version "3.1" 

CodeLocation M http://3 Community /code/groupware/latest 
SerialNumber "444-56-7777" 
35 SupportedFeatures "Virtual Chat Room" 

18 

SUBSTITUTE SHEET (RULE 26) 

BNSDOC1D: <WO 0043891A1_L> 



WO 00/43891 PCT/US00/01634 

5 

Make-Instance EmailService 
Name "Netscape Mail" 
Status "Active" 
Location "3Com Intranet" 
10 PhysicalAddress "Building300.FIoor2.Cube300" 

PhysicalMachine "Solaris" 
Owner "Rick Nottingham" 
Vendor "Netscape" 
Version "5.0" 

1 5 CodeLocation "http://3Community /code/email/latest 

Serial Number "456-34-6786" 

If middleware resides on a proxy device (rather than on the control device 200), the 
control device 200 will need to locate such a proxy service. An Operating System Service 
20 class can encode services such as a JVM Service, a Linux Service, or a Jini proxy. 



Make-Instance OperatingSystemService 

Name "Jini Proxy" 

Status "Active" 
25 Location "3Com Intranet" 

PhysicalAddress "Building300.FIoor2.Cubel20 n 

PhysicalMachine "Solaris" 

Owner "Rick Nottingham" 

Vendor "Sun" 
30 Version "5.0" 

RealTimeOS? "no" 



Finally, directory services 220 (such as the Jini Lookup) are modeled. This could be 
encoded in a class such as the Service Location Service. 

35 
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Make-Instance ServiceLocationService 



Name "Jini Lookup" 



Status "Active 



Location "3Com Intranet 1 



10 



PhysicalAddress "Building300.Floor2.Cubel50" 
PhysicalMachine "Solaris" 
Owner "Rick Nottingham" 



Vendor "Sun" 



Version "5.0 n 



15 



QueryProtocol "Jini Lookup Protocol" 
Schema "Service Directory Schema 1.0' 



The present invention integrates a number of services running on different machines, and 
provides control of the services from a thin platform with the a graphical user interface controlled 
by a console application adapted to the thin platform. A typical single station-based application is 

20 organized into a set of modular services running from different systems all networked together. In 
one embodiment described with reference to Figures 8-16 below, the Powerpoint application was 
chosen and a set of control services were developed to control the various aspects of presentation of 
a slide. A wide variety of other applications are also controlled using the present invention, and some 
examples are mentioned herein. The details provided below concerning the Powerpoint application 

25 will be modified in various implementations of the invention and for various applications other than 
Powerpoint, and are provided only as an example implementation. In this example, all these control 
services where then controlled from the a palm sized computer running the Palm computing 
environment. 



1. The PalmDOS (Palm Director}' of Services) and Powerpoint control program running on the 
Palm. 

35 2. PalmListener - The Java based server to integrate the Palm with the network. 



30 



In the example described, the system consists of the following parts: 



20 
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5 3. The Powerpoint control Services. 
4. The Jini lookup and registry service. 

All the services register themselves to the Jini registry service as soon as they are started. 
, When a user taps the PalmDOS service on the Palm screen, the PalmDOS service accepts the IP 

10 address of the PalrnListner as input and contacts it. Alternatively, a discovery message protocol can 
be exectured to automatically fmd the PalmListener. The PalmListener in turn contacts the Jini server 
and passes on to the PalmDOS a list of services that are available. When the user taps the Powerpoint 
Presentation Service on the Palm the PalmDOS contacts the PalmListener. The PalmListener then 
sends a stream of Palm executable bytes and the IP address/port details of the Powerpoint control 

15 service to the Palm. The PalmDOS service installs the Powerpoint control program (from the byte 
stream) and Launches it after passing the IP address/port details as a launch parameter. The 
Powerpoint control program then interacts with the Powerpoint control service to control the 
presentations. The Powerpoint control program offers a UI to control presentation slides in a limited 
way. 

20 

The PowerPoint control services is a collection of services which facilitate the control and 
display of a PowerPoint presentation in a networked environment. 

The PowerPoint control services aim to break down the various tasks in a typical single station based 
25 presentation of slides using PowerPoint application, into a set of modular services running from 
different systems all networked together. 

Some of the tasks that are performed by a presenter of a presentation include: 
_ Locating and loading a presentation from a storage subsystem into PowerPoint application. 
30 _ Once loaded, scroll through the presentation, one slide at a time, doing tasks like 

annotating/scribbling on the slide, along the presentation. 
_ Display the slides on a wide screen through a projector connected to the PC miming the 

PowerPoint application. 
_ Print the slides as needed. 

35 
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5 The PowerPoint control services simulate these tasks using the services shown in Figure 8, all 
interconnected among each other as needed: Console, Cpu, Storage, Display, Print. 
_ The presenter runs the Console Application 800, which through its interface lets him/her control 
all aspects of giving a presentation. The Console Application also controls the behaviour of other 
services. 

10 _ The Storage service 801 provides a listing of all presentations available to be presented, that are 
stored in the subsystem on which it is running. 
_ The Display service 802 provides the facility to display the slide images of a presentation on a 
display medium. 

The Print service 803 provides the facility to print the slide images of a presentation on a print 
15 medium. 

_ The Cpu service 804 provides the necessary interfaces to communicate with the PowerPoint 
application to actually facilitate loading a presentation and showing/printing the slide images on 
the connected display and print services. 

20 The PalmDOS and Powerpoint control program run on the Palm. The PalmDOS application 

basically connects to the PalmListener and gets a list of service and displays it to the* user. When the 
user taps Powerpoint Control the PalmDOS downloads the executable version of the Powerpoint 
Control program and installs it on the Palm. It then launches the Powerpoint control program after 
passing the IP address /port details of all the powerpoint presentation services. ThePowerpoint 

25 control program presents a user interface for the user to connect to and control the presentation slides 
present in the Storage service and display it in the Display Service.The Palm does all network 
communication by means of establishing a Point to Point Connection Protocol (PPP) connection with 
a proxy. TCP/IP packets are sent over the PPP. A more detailed explanation of the Powerpoint 
control program is provided later. Please refer the appendix for details about the PPP connection 

30 between the Palm and Windows 95 and Windows NT machine. 

General Overview Jini technology: 

The JINI registry service was built using the Sun Microsystems Jini Technology. Jini 
35 Technology is a breakthrough initiative based on JavaTM technology that enables all types of devices 
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5 to simply connect into impromptu networks, making access and delivery of new network services as 
simple as plugging in a telephone. Built on top of a Java software infrastructure, Jini technology 
enables all types of digital devices to work together in a community put together without extensive 
planning, installation, or human intervention. Each device provides services that other devices in the 
community may use. These devices provide their own user or programmatic interfaces, which ensures 
10 reliability and compatibility. 

The lookup service in Jini technology is a lightweight, but very powerful repository of 
services. Its structure is unique in that it uses the Java platform type system as the namespace. This 
means it does not store fixed name-value pairs, but objects and object graphs — the actual behavior 
15 of an object. This has two distinct advantages in a distributed system. First, you can search for an 
object or service based on a desired behavior, not just on its name. Second, once you find an object 
in the directory, you immediately know how to use it . 

Environment 



20 The services are developed as Java classes in one embodiment, enabling them to be run on any 
Java supported platform. 

Appropriate JNI interfaces enable the controlling of PowerPoint and other platform specific 
applications from a Java class. Tools for development and test of the Powerpoint example 
described herein are set forth in the following table. 

25 

J. Development & Test Environment 



j Software 


Purpose 


Visual Cafe3.0JDK1.2 


Java classes for the various services. 


PowerPoint and Office 97with 
Service Pack 1 


Application controlled by the Cpu service. 


Visual Studio 6.0 


JNI interfaces on the PC. 
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Annltr^tinn that thp ^torapp service denend*; on 
when running on the PC. 


Internet Explorer 4.0 


Application that the display service depends on 
when running on the PC. 



5 

The PalmDOS application is mainly used to connect with the PalmListner and get a list of 
services and present to the user. Once the user taps on the service desired, the PalmDOS 
function downloads a byte stream containing a Palm executable file of Powerpoint console 
program. It also gets a List of addresses to the locations of the various Powerpoint Control 



10 services. 

As soon as the palmDOS service obtains the Byte stream of the Powerpoint control program 
application it installs the Powerpoint control program on the palm and launches it. It also 
Passes the list of addresses as a command line argument to the Powerpoint control program. 

15 Once the user enters the IP address of the PalmListener service in the field provided and taps 
the OK button the PalmDOS connects to port 2300 (or some other specified part number) of 
the PalmListener. Which in turn spawns a child to handle the PalmDOS request. The 
communications are handled in such a manner that for each request the PalmDOS creates a 
new socket and reconnects to port 2300 of the PalmListener. This design helps the 

20 PalmListener to manage multiple Palms connecting to it at the same time. 

The Powerpoint control program Application is the main interface through which the collection 
of services is controlled to facilitate giving a presentation. It offers the user the list of the 
various service locations and the user can choose from the list. 

25 2.0.1 Command :Get a list of Services registered to the JINI 

The following command is issued by the PalmDOS to the PalmListener to get a list of services. 
SENDRQ 

30 2.0.2 Command : Download Powerpoint control program Byte stream 
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The following command is issued by the PalmDOS to the PalmListener to get the Powerpoint 
control program byte streams and the list of addresses of the powerpoint control services. 
Using the touch screen on the palm, the user executes a : 

Tap on the name of the service on the List 



0043891 A 1 ! > 
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15 



25 



Console Application 

The Console Application is the main interface through which the collection of services are 
controlled to facilitate giving a presentation. Console Application is the only service in the 
collection that has an user interface. All the other services may provide minimal user interface 
required to setup the programs. 

Since the Console Application is expected to run from a palm sized computer, such as the Palm 
Computing Environment available from 3Com Corporation, the console application service is 
expected to use least number of resources in terms of network connections and features like 
views and windows. 



The Console Application functionality can be described using the following main tasks: 

1 . User interface 

2. Collection of information about the location of various services that will participate in the 
presentation. 

20 3. Communication with the Cpu service to carry out the presenter's commands. 

4. Opening a presentation and navigation of slides in a presentation: 

5. Editing/Annotating/Scribbling on a slide during the presentation. 

6. Display/Print the image for current slide on a connected Display/Print service. 



K User Interface 



The Console Application is the only service in this example which has an user interface. It is 
started in this example by selecting the PalmDOS icon in the screen of Figure 9. 
There are three distinct stages to the user interface: 
_ Collection of information about service location 
30 _ Navigation of slides 

Editing/ Annotating/Scribbling on a slide 



L. Service Location 
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5 When started, the Console Application presents the user with a choice of system addresses (IP 
address/host name & tcp port no) where the various services are configured to run. It can 
connect to a single Cpu service and a single Storage Service and multiple Display and Print 
services. The PalmDOS program then shows the screen of Figure 10 in one example. 

10 In one implementation, the location service provides a hardwired list of system names with 
appropriate port numbers where the various services are expected to be configured and run 
(Fig. 1 1). In another implementation, the user interface on the palm sized computer, or other 
console application host, provides an icon, which when selected invokes a discovery message 
protocol to access a registry of available services, and produce the list based upon information 

15 gathered at the site served by the registry which responds to the discovery message. The 
PalmDOS application provides the system addresses as command line parameters when the 
Powerpoinl console application is launched. 

M. Cpu Communication 

20 Once the various hosts/locations have been selected, Console Application connects to the Cpu 
service and informs it about the various other choices. Even though the Console lists locations 
for all available Storage/Display /Print services, it does not directly communicate with any of 
them. Instead, it lets the Cpu service establish links to the selected services and communicate 
with them as needed based on the task to be performed. 

25 

Once connected, the Cpu service in turn tries to connect to the Storage, Display and Print 
services. The list of presentations available on the storage subsystem is then sent back to the 
Console to be displayed. The user can select a presentation from the list, that needs to be 
presented. 

30 

The Console remains connected to the Cpu until all the services are stopped. The Cpu service 
acts as a gateway between the Console and all other connected services. 
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5 Most of the commands that are used in this example have one or two letter codes with optional 
parameters following the command code. The command typically results in appropriate 
response. 



10 

2.1.1 Command: Location of Display Service 

The following command is issued by the Powerpoint control program Application to Cpu, to 
specify the location a Display service: 

15 P < hostname > ; < portno > 

Where p is the command code, < hostname > is the name of the host where the display service 
is running and <port> is the port no where the display service is bound. 

20 On receipt of this command, the Cpu service establishes a socket connection to the display 
service at the given location. 

2.1.2 Command: Location of Storage Service 

The following command is issued by the Powerpoint control program Application to Cpu, to 
25 specify the location of the Storage service: 

1 < hostname >;< portno > 

Where 1 is the command code, < hostname > is the name of the host were the storage service 
30 is running and <port> is the port no where the storage service is bound. 

On receipt of this command, the Cpu service establishes a socket connection to the storage 
service at the given location. 
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5 After establishing the connection, Cpu sends the following command to the connected storage 
service: 

1 

10 where 1 is the command code which lists all files known to the storage service. The storage 
service lists the file names with appropriate url names as configured with the web server 
which is running along side the storage service. 

The list of presentation names is then returned back to Powerpoint control program to be 
15 displayed. 

2.1»3 Command: Location of Print 

The following command is issued by the Powerpoint control program Application to cpu, to 
specify the location of the Print service: 

20 

Ph < hostname > ; < portno > 

Where Ph is the command code, < hostname > is the name of the host where the print service 
is running and <port> is the port no where the pritn service is bound. 

25 

On receipt of this command, the Cpu service establishes a socket connection to the print 
service. 

2.1,4 Command: Print current slide 

The following command is issued by the Powerpoint control program Application to Cpu, to 
30 print the current slide (of a presentation or blank page which is being scribbled on): 

Ps 
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This command must be given only after establishing a connection to the print service. On 
receipt of this command, the Cpu service copies the image of the current slide to the print 
service which will then using IE, print the image on currently selected printer. 

2.1.5 Command: Print all slides 

The following command is issued by the Powerpoint control program Application to Cpu, to 
print all slides in current presentation: 



Pa 

As with Ps, this command must be given only after establishing connection with print service. 
This command is valid only when a presentation is open and not in blank slide mode. 



2.1.6 Command: Open Presentation 

The following command is issued by the Powerpoint control program Application to Cpu, to 
20 open a presentation. The name of the presentation is selected from the list of names returned 
earlier after locating the storage service. 

o < presentation > 

25 

2.1.7 Command: Close Presentation 

The following command is issued by the Powerpoint control program Application to Cpu, to 
close an open presentation: 



c 

30 

This is done so that at the maximum only two presentations will be open at any time (the 
presentation opened explicitly and the scratch presentation used for blank screen). 
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2.1.8 Command: Save Presentation 

During the presentation, if changes are made to a presentation, it is possible to save the 
changes back on the storage. The save presentation could also be used to save the scribbled 
slides. 

Clicking the Save button, instructs the Cpu service to save the current presentation on the 
10 storage service. If the current mode is blank screen mode, then the scribbled blank slides are 
saved on the storage service (using the timestamp as file name; like mm-dd- 
y yy y . hh . mm . ss . ppt) . 

The syntax for the save command as sent to the Cpu service is: 

15 

v 

This Cpu responds to this command with the Saved message. Depending on the current mode, 
the appropriate presentation is saved by the storage service. 

20 

2.1.9 Command: Get Titles 

After opening a presentation, the Powerpoint control program Application gets a list of titles 
from the current presentation. This is achieved in two steps: 

25 Get title count 

Get title for a given slide 

2.1.9.1 Command: Get title count 

To get the number of titles that are there in a presentation,Powerpoint control program sends 
30 the following command to Cpu: 

Gc 
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5 This results in a count, which is same as the number of slides in the presentation. 
2. 1.9.2 Command: Get title for a given slide 

To get the title for a specific slide in the current presentation, Powerpoint control program 
sends the following command to Cpu: 

10 G<idx> 

Where < idx> is the slide no (which is between 1 and last slide no). 

2.1.10 Command: Get Text 

15 Whenever a slide is navigated to (either by clicking on a title, or using the navigation buttons), 
the Powerpoint control program Application gets the text from the current slide and displays it. 
This is done so that the view from the presenter is in sync with what is displayed at all the 
connected display services. 

20 Like the get title command, this command is also done in two steps: 

Get text count 

Get text for given text box 

25 

2.1.10.1 Command: Get text count 

To get the no of text boxes in current slide, Powerpoint control program sends the following 
command to Cpu: 

gc 

30 

This returns the count of text boxes; in most slides, this will be 2; 1 for the title and other for 
the outline body text. 

2.1.10.2 Command: Get text for a given text box 
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5 To get the text from a given text box, Powerpoint control program sends the following 
command to Cpu: 

g<idx> 

10 This returns the text for the given text box. In most slides, there will be only 2 text boxes 

(actually, in PowerPoint object model lingo, these are referred to as placeholders, one being a 
title placeholder and other being text (outline) placeholder). 

Cpu formats the body text, if indentation-using bullets are present. To show the outline of text 
15 properly for various indentation levels, Cpu uses the following convention: 

< level 1 indent > 

< level 2 indent, with 2 leading spaces > 
+ < level 3 indent, with 4 leading spaces > 
* < level 4 indent, with 6 leading spaces > 

20 > < level 5 indent, with 8 leading spaces > . 

1.0.1 Command: Get speaker notes 

To get the text of speaker notes, which are part of the current slide, Powerpoint control 
program sends the following command to Cpu: 

25 

N 

This returns the speaker notes (if found). No special formatting is done to the text found as 
speaker notes. This text is not editable. 

30 

1.1 Slide Navigation in PowerPoint control program application 

Once the user opens a presentation (from the list of presentations of Fig. 15 for example), the 
Powerpoint control program sends a command to Cpu service to open the presentation. This 
sets off a sequence of commands that finally results in the presentation being loaded in to 
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PowerPoint and the first slide shown on all displays. During the presentation, the Powerpoint 
control program remains in sync with what is shown on the displays. 

The Powerpoint control program shows only the text from the current slide, whereas the 
display shows the complete image as would be shown if the presentation were given directly 
from inside PowerPoint. 

The Powerpoint control program provides controls for easy navigation of slides in a 
presentation: 

_ A list of titles in the current presentation is shown in a separate Form like Fig. 16 for 
example. Clicking on any slide, results in a command to Cpu to go to that slide. This in 
turn results in the image for the slide shown on all connected displays. The window, which 
shows the text of the current slide on the Powerpoint control program, is also updated with 
the text from the just navigated slide. 

_ Buttons are provided to go to: first slide, last slide, next slide and previous slide. 

2.0A Command: Goto slide using title 

When the user selects a title to scroll to, the Powerpoint control program Application issues the 
following command to Cpu service: 

t < title-text > 

where < title-text > is the title as it appears on a slide. If a slide does not have a title, the is 
substituted for the title. 

2.0»2 Command: Goto first slide 

When the user clicks this button, the Powerpoint control program issues the following 
command to cpu service: 
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5 < < 

Cpu service makes the first slide as the current slide and exports the image to display 
services. Powerpoint control program gets the text for the first slide and shows it. 

2.0.3 Command: Goto next slide 
10 When the user clicks this button, the Powerpoint control program issues the following 
command to cpu service: 

> 

15 Cpu service moves to the next slide in the presentation. If already in the last slide, then nothing 
happens. 

2.0.4 Command: Goto previous slide 

When the user clicks this button, the Powerpoint control program issues the following 
20 command to cpu service: 

< 

Cpu service moves to the previous slide in the presentation. If already in the first slide, then 
25 nothing happens. 

2.0.5 Command: Goto last slide 

When the user clicks this button, the Powerpoint control program issues the following 
command to cpu service: 

30 

> > 

Cpu service moves to the last slide in the presentation. 

35 

SUBSTITUTE SHEET (RULE 26) 



BNSDOCID: <WO 0O43891A1J_> 



WO 00/43891 



PCT/USOO/01634 



2.1 Edit Text/ Scribble on a slide 

After opening a presentation and during navigating the slides, the user may do the following 
tasks on the current slide: 

Edit the text on the slide. 
_ Scribble on a blank slide. 

2.0.1 Edit text 

During the presentation, the user can change the text on the current slide. An Apply Menu item 
is provided which when clicked, sends a command to the Cpu to apply the text changes to the 
current slide. As with navigation of slides, when any changes are made to the slide, the Cpu 
instructs the Display service to display the changed slide's image. 

While editing the text, the user can easily simulate the indented (bulleted) structure of body 
text using the same notation that Cpu uses to illustrate the outline text. Unlike the formatting 
done in Cpu, in the Powerpoint control program, no additional leading space needs to be 
specified; just the appropriate first letter to signify the indentation (bullet) level. 

Level 1 indent 

Level 2 indent 
+ Level 3 indent 
* Level 4 indent 
> Level 5 indent 

3.0.0.1 Command: Change the text on slide 

The Powerpoint control program Application shows the text from various text boxes in the 
slide as strings in a text area, as in Fig. 12 for example. The text from each text box is 
delimited by tilde character C). The user can change the text within the delimiter and apply the 
changes onto the slide using pull down menus, like shown in Fig. 13. 
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5 

When the user modifies the text in a slide, the Powerpoint control program issues the following 
command: 

S < idx > ; < slide-content-text > 

10 The slide content text is the entire text for the slide as changed by the user. The Cpu service in 
turn applies the changes on to the slide. This command will only change the text on the slide; it 
will not be exported into an image and shown on displays; To do that the Apply command 
needs to be issued. 

15 The <idx> parameter can have one of two values: if set to 1, the following text includes the 
title text for the slide and the outline body text. These two texts are separated by f) character; 
if set to 2 , the following text is the changed text for the speaker notes. 



3.0.0.2 Command: Apply changes and refresh display 

20 After making the changes, to apply the changes on to the slide and show it on displays, the 
Powerpoint control program issues the following command: 

A 

25 This instructs the Cpu service to apply all the changes on to the slide and export the image onto 
all connected display services. 

3.0.1 Scribble 

During the presentation, the user can request for a blank slide and scribble on it using free 
form strokes on a predefined area of the Powerpoint control program Application's window as 
30 shown in Fig. 14. A Blank menu item is provided which toggles the blank slide and normal 
slide mode. 

Clicking the Blank menu item, instructs the Cpu service to use a blank slide and show it on the 
Display services. In sync with the Cpu and display services, the Powerpoint control program 
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5 also shows a blank painting area. The presenter is then allowed to scribble on this area and the 
strokes are conveyed to the Cpu and in turn to the display services. Clicking on the Unblank 
button toggles the mode back to text mode and the last slide that was being shown is switched 
back as the current slide (the Cpu and display services also switch their contexts to the correct 
slide). 

10 

3.0.1.1 Command: Scribble 

The syntax fpr the scribble command as sent to the Cpu service is: 
s < cnt-nodes > ; < xynodel > < xynode2 > ... 

15 

where the first parameter is the number of nodes in the scribble and the actual nodes 
represented as x and y coordinates. 

3*6 Functional Flow of the Powerpoint Control Program 

20 

There are five different forms in the Powerpoint Control program. Each form has a handler to 
handle the events generated in that form. Following are the forms and their corresponding 
handlers. 

25 Main Form (showing the list of addresses): MainFonnHandleEvent 
Title Form (showing the text for the slide): TilteFormHandleEvent 
Title Table form(showing the list of titles) : TitletableHandleEvent 
File Form(showing the list of files) : FileFormHandleEvent 
Scribble Form(bIank form to scribble): ScribbleFormHandleEvent 

30 



35 



TitleFonnlnit is called ti initialise the Title form and is called whenever the Title Form is re- 
entered. GetText is called when the navigation buttons are pressed. The function GetTime is 
called whenever the clock field is to be updated. 
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5 The Send and Recv funcations are called whenever a network send and receive is required. 

Finally the StopApplication is called to close all the sockets and the network library. 



10 



15 



3 Command Summary 



3.1 PalmDOS Command Summary 



Command 


Description 


SENDRQ 


Send the list of services currently registered 


Powerpoint Control Services 


Send the byte stream of Powerpoint control program 
application and the list of addresses of the Powerpoint 
Control Services 



20 



3.2 Console Command Summary 



25 



The following table summarizes the commands that are received by the Cpu service from the 
Powerpoint control program Application; these commands have been briefly explained in the 
section on Powerpoint control program Application: 



Command 


Description 


p < hostname >;< port > 


Display service host and port address 


1 < hostname > ; < port > 


Storage service host and port address 
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Ph < hostname > ; < port > 


Print service host and port address 


rS 


Print current slide 




Print all slides (not blanks) 


o < url -presentation > 


Open a presentation given the URL 


c 


Close currently open presentation 


V 


Save current presentation 


LjC,U < luX > 


Get titles from current presentation 


gc, g < idx > 


Get text for current slide 


o < idx > ; < text > 


Set text for current slide 


A 

A 


Apply changes to slide and refresh 


t < title > 


Scroll to slide with specified title 


< < 


Scroll to first slide 


> > 


Scroll to last slide 


< 


Scroll to previous slide 




ocroii 10 next sucic 


b 


Toggle blank screen mode 


s < nodes > ; < nodelist > 


Scribble on blank screen using nodes 


Q 


Quit 



5 



Cpu Service 

10 

The Cpu service is the main gateway interface between Console and the other services like 
Storage, Display and Print. The Console does not directly communicate with Storage, Print 
and Display services; only through the Cpu service. 
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5 In this example for controlling the PowerPoint application and Office 97, the Cpu service is 
from a Windows 95/NT workstation. Also this service would require Java Native Interface 
(JNI) to another OLE automation module which actually would communicate with PowerPoint. 

The Cpu service functionality can be explained based on the following tasks it performs: 
10 1. Communication with Console Application 

2. Communication with Storage service 

3. Communication with Display service 

4. Communication with Print service 

5. JNI to PowerPoint automation module 

15 

N, Communication with Console Application 

The Console sends commands to the Cpu service which responds to the commands and sends 
responses back. Almost all commands result in some form of response. 

20 After startup, the first set of commands from the Console to the Cpu, establish the location of 
the various services like Storage and Display. Since the Console does not communicate directly 
with other services, Cpu establishes connection to these services and passes appropriate 
response back to Console. 

25 The Console sends the following commands to the Cpu. Some of these commands need to 

given in a sequence (like navigation commands follow the open presentation command, which 
follows service location messages, etc.) 



_ Set service location for Display, Storage and Print services 

30 _ Get a list of presentations from Storage 

_ Open a presentation 

_ Get a list of titles in currently open presentation 

_ Go to a slide using the title 

_ Go to the first, next, previous or last slide 
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5 Get the text content from current slide 

_ Apply changes to text on the current slide 

_ Add notes to current slide at given location 

_ Enable/Disable blank screen mode 

_ Apply scribble on current blank slide 

10 _ Display current slide on Display services 

_ Print current slide on Print services 

_ Shutdown 



O. The Communication with Storage service 

15 The Cpu service gets a list of presentations available on the storage subsystem and sends them 
back to the Console. 

Even though the storage service makes a list of presentations available on the storage 
subsystem, the Cpu service is responsible for actually transferring the file from the storage 
20 sub-system to the machine where the Cpu service is running. 

Storage Command Summary 

The following table summarizes the commands that are issued by the Cpu service to the storage 
service: 

25 



Command 


Description 


1 


Return list of presentations on storage 


q 


Quit 



P. Communication with Display service 
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5 The Cpu service creates images of slides as they are being scrolled through and sends them to 
the display service to be shown on any connected display medium (like Video screen, projector 
etc.) 

The communication between Cpu and Display is mostly one way and include: 
10 _ Cpu sends an image to display; Display shows it on the display medium. Multiple Display 
services could be connected to the Cpu, which would all receive images to be shown. 
_ Shutdown 

Display Command Summary 
15 The following table summarizes the commands that are issued by the Cpu service to the 
Display service: 



Command 


Description 


<imagesize> <data> 


Store the image and show it on display 


f 


Toggle full screen mode 


q 


Quit 



Q. Communication with Print service 
20 The print service is very similar to the Display service, the only difference being the output 
medium. 



R. JNI to PowerPoint automation module 

The PowerPoint automation module is responsible to translate the commands from Console, 
25 carry them out on the PowerPoint application. 

These commands include: 

_ Open a presentation using the given local file name 
_ Close an open presentation 
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5 _ Get a list of titles in current presentation 

_ Get the number of slides on current presentation 
_ Goto a slide based on the given title 
_ Goto a slide based on the slide no 
_ Goto first, next, previous or last slide 
10 _ Get text from all textboxes from current slide 
_ Apply text changes on current slide 
_ Add notes to current slide at given location 

_ Switch presentations between current open presentation and a temporary scratch 
presentation 

15 _ Switch between current slide on an open presentation to a blank slide on scratch 
presentation 
_ Apply scribble on a blank slide 

_ Create and export the image for current slide (from open presentation as well as from 
scratch) to some location 

20 

S. JniCpu class 

The following code segment illustrates the typical implementation for the JNI for PowerPoint 
automation module. 

25 

COPYRIGHT 3COM CORPORATION 2000 

public class Cpu 

{ 

30 /** 

* StartPPO 

* Stan the PowerPoint application. PowerPoint is a single instance application (only one 

* copy of it can be ninning on the system at a time. If an instance is already running, then it 
35 * will be used. Else a new instance is created and made visible. The instance is made visible 

* so thai some of the scribble and export functions work properly. 

* If this function fails, then rest of the methods can't be used. 
* 

40 * Input: 

* Nothing 
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* Output: 

* Nothing; PowerPoint is started. 
* 

* Returns: 

10 * 0 - If PP started succesfully. 

* ! = 0 - Error codes, that explain the failure. 
*/ 

public native long StartPP (); 

15 /** 

* StopPP () 

* Quit the PP application. If any presentations are open, they are all closed. 
* 

20 * Inputs: 

* Nothing 

* Outputs: 

* Nothing 
* 

* Returns: 

* 0 - If PP stopped succesfully. 



* ! — 0 - Error codes, that explain the failure. 
*/ 

30 public native long StopPP (); 
/** 

* OpenPresentation () 

35 * Open a presentation. The file name given as input to this method is an absolute path that must 

* be accessible to the PP. This implementation only supports one open presentation at a time. If 

* you need to open another presentation, then first close the currently open presentation using the 

* ClosePresentation method and then invoke this method. 

40 * Input: 

* strFile - Name of the presentation (.ppt file) to open 
* 



* Output: 

* Nothing 



* Returns: 

* 0 - If given presentation opened 

* ! — 0 - Error codes, that explain the failure. 
*/ 

50 public native long OpenPresentation (String strFile); 



* GetExponFile () 

55 * Get the name of the export file (as a result of a command mat results in a export of a slide 

* commands like scroll page, add notes, add scribble result in export). 



* Input: 

* Nothing 
# 

* Output: 
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5 * Nothing 

* Returns: 

* Name of the file, last exported. 
*/ 

10 public native String GetExportFile (); 



/ 

Close Presentation () 

Close the currently open presentation. 

Input: 

Nothing 

Output: 
Nothing 

Returns: 

0 - If presentation closed. 

public native long ClosePresentation (); 



15 



20 



25 



35 



55 



* GetCurrentSlide () 

30 

* Get the current slide no in the currently open presentation. 



* Input 

* Nothing 

* Output 

* Nothing 



* Returns: 

40 * < no > - Current slide no 1 - max slide no 

* -1 - if a presentation is not open or other failure 
*/ 

public native long GetCurrentSlide (); 

45 /** 

* GetSlideCount () 

* Get the number of slides in the currently open presentation. 

50 * Input: 

* Nothing 



* Output: 

* Nothing 



* Returns: 

* <cnt> - No of slides in current presentation 

* -1 - If a presentation is not open, or other failure 
*/ 

60 public native long GetSlideCount (); 
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/** 

* GetTitles () 
* 

* Get all the titles from the currently open presentation. This is a list of lines separated by 

* *\n\ If a slide does not have a title, then a blank line is returned. 

* Input: 

* Nothing 
* 

* Output: 

15 * Nothing 

* Returns: 

* String - Contains the titles. 
*/ 

20 public native String GetTitles (); 
/** 

* GetText () 
* 

25 * Get all the text from the currently open presentation. This is a list of lines separated by 
* 

* Input: 

* Nothing 

30 

* Output; 

* Nothing 
* 

* Returns: 

35 * String - Contains the text. 
*/ 

public native String GetText (); 
/** 

40 * SetText () 

* Set the changed text on the current slide on the currently open presentation. This is a list of lines 

* separated by ? \n*. 
* 

45 * Input: 

* strText - String containing the changed text 
* 

* Output: 

* Nothing 

50 

* Returns: 

* 0 - If change applied properly 

* ! = 0 - Error codes, explaining the failure. 
*/ 

55 public native long SetText (String strText); 
/** 

* GotoSlideNo () 

60 * Scroll to the given slide no. The no must be between 1 and max-slides in the presentation. If any display 

* services are registerd with the CPU service, then the CPU service would scroll to this slide and export it 
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3 * and push the URL to the display services. 

* Input: 

* iNo - Slide no to scroll to. 

1 0 * Output: 

* Nothing 

* Returns: 

* 0 Scrolled to the given slide successfully. 
15 * ! = 0 - Error codes explaining the failure. 

*/ 

public native long GotoSIideNo (int iNo); 
/** 

20 * GotoSlideDir () 
* 

* Scroll in the given direction. Export the slide and push the URL to all known display services. 

* Input: 

25 * iDir - -1 scrolIPrev 

* - -2 scrollNext 

* - -3 scrollFirst 

* - -4 scrollLast 

* - -5 scrollCur - refresh the current slide 



30 



50 



* Output: 

* Nothing 



* Returns: 

35 * 0 - Scrolled to the given slide successfully. 

* ! = 0 - Error codes explaining the failure. 
*/ 

public native long GotoSlideDir (int iDir); 

40 /** 

* GotoSlideTitle () 

* Scroll to the given title. Figure out the slide no based on die text. 
* 

45 * Input: 

* strTitle - Tide of the slide to scroll to. 



* Output: 

* Nothing 



* Returns: 

* 0 - Scrolled to the given slide successfully. 

* ! = 0 - Error codes explaining the failure. 
*/ 

55 public native long GotoSlideTitle (String strTitle); 
/** 

* BlankScreen () 

60 * Turn On/Off the blank screen mode. This will create a new blank slide on a scratch presentation 

* that is used to do the scribble. 

48 

SUBSTITUTE SHEET (RULE 25) 



RNSDOCtD: <WO 0043891 A1 1 > 



WO 00/43891 



PCT/US00/01634 



5 

* Input: 

* bFlag - true - Blank screen 

* false - UnBlank screen 

10 * Output: 

* Nothing 

* Returns: 

* 0 - A blank slide created successfully. 
15 * ! = 0 - Error codes explaining the failure. 

*/ 

public native long BlankScreen (boolean bFlag); 
/** 

20 * AddScribble () 
* 

* Scribble on current blank slide using the give nodes. Each node represents a point on the slide. A line 

* is drawn from one point to the next till all nodes are drawn. The blank screen mode must have been 

* turned ON before making this call. After drawing the nodes on the slide, the page will be exported out * and 

25 pushed out on to the known display services. 
* 

* Input: 

* LNodes - No of nodes 

* strNodes- Nodes represented as a pair like "(xxxx^yyyy)*' 

30 

* Output: 

* Nothing 
* 

* Returns: 

35 * 0 - If scribble added to the current slide successfully 

* 1=0- Error code explaining the failure 
*/ 

public native long AddScribble (long INodes, String strNodes); 

40 /** 

* AddNotes () 
* 

* Add a sticky notes to current slide at given location. If there are other notes at the 

* current location, men PP will automatically slide the notes down. 

45 

* Input: 

* x,y - Location where the notes is to be shown. 

* strNotes- String containing the notes. 

50 * Output: 

* Nothing 
* 

* Returns: 

* 0 - If notes added successfully 

55 * ! = 0 - Error codes explaining the failure 
*/ 

public native long AddNotes (int x, int v. String strNotes); 

} 
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5 

T. CPowerPoint class 

The following code segment illustrates the typical implementation for a PowerPoint automation 
class as implemented in C+ -K As can be seen, there is a direct one to one correlation 
between the JniCpu class and CPowerPoint class. 



10 COPYRIGHT 3COM CORPORATION 2000 
class CPowerPoint 

{ 

public: 

CPowerPoint(); 
15 virtual — C Power Point(); 

public: 

// Stan an instance of PowerPoint (if one already running, then use that) 
// since PP is a single instance app 
20 intStartPPO; 

// Stop (File/Exit) running PP 
int StopPP (); 

25 // make PP visible 

int Visible (bool bVisible); 

I J open given presentation 

int OpenPresentation(_bstr t bstrFile); 

30 

// return the no of slides in presentation 
int SlideCount (); 

// return list of titles in presentation 
35 int GetTitles ( bstrj *pbstrTitles); 

// goto a given slide (based on direction) 
int GotoSlide (enura ScrollValue dir); 

40 // goto a given slide (absolute slide no) 

int GotoSlide (int iNo); 

// goto a given slide (based on slide title) 
int GotoSlide (_bstr_t bstrTitle); 

45 

// add notes to current slide 

int AddNotes(_bstr_t bstrNotes, float cX, float cY); 

// add scribble to current slide/new blank slide 
50 int AddScribble (bool fNewSlide, long INodes, struct Point *pNodes); 

// get text from current slide 
int GetText ( J>str_t *pbstrText); 

55 // set the new text 
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5 int SetText (_bstr_t bstrText); 

// get the size of the slides (width & height, in points) 
int GetSize (int *pWidth. int *pHeight); 

10 // Export current slide to jpg file 

int ExportO; 

int BlankScreen (); 

15 // close opened presentation 

int Close Presentation (); 



20 



30 



50 



// name of recently exported file 
bstr t mbstrExportFile; 



long CurrentSHde () { return m fBlank = = false ? mJSlideldx : 1; } 
long FileSearch (_bstr_t bstrPath); 
25 FoundFilesPtr FoundFiles () { return m_pPP- > GetFiIeSearch()- > FoundFiles; } 



private: 

// status flags 
cppFlags m cppFlags; 

// main PowerPoint application pointer 
// use this to create an instance of PP 
_ApplicationPir m_pPP; 

35 // currently open presentatio pointer 

// only one presentation is active and open at a time (for now) 
PresentationPtr m_pPres; 

PresentationPtr m_pBlankPres; 
40 bool mfBlank; 

// cache slide titles 
bstr t *m_bstrTitles; 

45 // no of slides in pres 

long mJSHdeCnt; 

// current slide idx 
long mJSlideldx; 

// refresh titles 
void RefreshTitles (); 



55 emim ScrollValue { 

scrollFirst = 1, 
scrollNext = 2, 
scrollPrev = 3, 
scrollLast = 4 

60 }; 
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Storage Service 



The Storage service is a gateway interface between the Cpu service and a web server. The 
Storage service is only responsible to collect a list of presentations available on the subsystem 
10 shared by the web server. It is not required to actually transfer the presentations to the Cpu 
service. That is done in coordination with the web server. 

After starting up, the Storage service gets commands from the Cpu service, which include: 
_ Get a list of presentations on the storage subsystem and return the list in the form of URLs 
15 _ Shutdown 

The URLs returned by the Storage must be created in such a way that when the Cpu service 
needs to load a presentation it is able to download the file from the web server which is also 
running on the same system. 

20 

Display Service 

The Display service is an output only type of service, in the sense that it does not communicate 
back results to any other service. Many instances of the Display services may be running on 
25 different systems, all of which will be connected to the Cpu service. The purpose of this 
service is show the image on a display medium, when instructed to do so. 

After starting up, the Display service gets commands from the Cpu service, which include: 
_ Copy the image for a slide and show it on the display medium 
30 Shutdown 
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5 Whenever the context requires, Cpu service generates the image for the current slide being 
presented and instructs the Display services to refresh the display about the new image. The 
Cpu sends the entire image to the Display service which shows the image. 

Print Service 



10 The Print service is an output only type of service and is very similar to the Display service, 
the only difference being the output medium. Unlike the Display service, the Cpu service will 
not refresh the image often: it will do so only on instruction from the Console Application that 
a print image is requested of the current slide. 

15 When required, the Print service will print the image on the connected print medium. 

After starting up, the Print service gets commands from the Cpu service, which include; 
_ Copy the image for a slide and print it on the print medium 
_ Shutdown 

20 



U. Variations on the Network-based Control Application 

Several variations of the general control paradigm can be defined: 
The middleware (Jini, in our example) may not be physically resident on the control device 

25 200. In this case, a proxy is used which runs middleware on behalf of the control device 

200. Functionally speaking, the same design will still apply. One change to the design can 
be used to support a modified GUI for the control device. In these embodiments, the GUI 
program is modified by the proxy device to account for specific limitations of the control 
device. Also, in some embodiments, the control device may include preset applications that 

30 interface directly with the proxy device. Such an architecture would support a limited set of 

network services but would likely result in very small and lightweight applications on the 
control device. 
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5 _ The control paradigm is not limited to palm sized computers. Any computing device with 
restricted computing power could be used as a control device 200 for any network-based 
resource. Switches, hubs, routers, and other networking devices are candidates for a 
control device 200. The network-based resources they use could include any service that 
cannot physically reside on the networking device due to restrictions such as limited 
10 memory. 

_ Some embodiments of the invention include only the control device and the programs on 
the control device; other embodiments include some and/or all of the programs in computer 
readable media, or in electromagnetic waveforms. 

15 

V. Other Applications 

Integrated desktop/Palm email access. E.g., use the Palm to access the desktop's 
Outlook program to view/print an attachment. 

_ Calendar access and update: group calendaring and contacts information access. 
Integrate the Palm calendar/contacts with the desktop or enterprise calendar/contact 
20 systems. For example, the Palm may have only a subset of the enterprise's phone 

list, the Jini interface may support an expanded search (e.g. via an LDAP server) to 
search enterprise wide contacts. 

_ Substitute user interfaces: provide user interfaces to devices that do not have any, or 
good, user interfaces, e.g., hearing aid adjustment (note potential prior art that 
25 Elaine had mentioned), control system devices. The Palm could control network 

device configurations (e.g., switch/router), e.g., provide a telnet client in the Jini 
repository to allow the Palm to change a network device's configuration. Also, 
home network remote control is possible. This provides ease of use for controlling 
home network devices, gateways, home appliances, TVs etc. 

30 

_ Desktop access/control: PC Any where like access that is controlled from the Palm. 
This could allow basic editing of documents, which maintains desktop application 
qualities of the documents. For example, use the Palm to fill out a Word document 
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form (Word is running on a desktop). This maintains the Word nature of the 
completed form. Thus, the user does not have to translate between document types. 

_ Typical network services: printing, faxing. 

_ Web access control: control web surfing. 

_ Group settings/management: e.g., controlling video conferencing/photo capture, 
controlling a shared electronic whiteboard, conference room reservation. 

_ Language and/or speech translation. 
W. Conclusion 

The foregoing description of various embodiments of the invention has been presented 
for purposes of illustration and description. It is not intended to limit the invention to the 
precise forms disclosed. Many modifications and equivalent arrangements will be apparent. 
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CLAIMS 

What is claimed is: 

1 . A method of controlling a service on a network using a palm sized computer, 
the palm sized computer being coupled in communications with the network, the 
method comprising: 

accessing a description of the service from a directory of services, the description of the 
service including at least a reference to program code for controlling the service; 

downloading the program code to the palm sized computer; 

the palm sized computer executing at least a portion of the program code; and 

sending control commands to the service from the palm sized computer in response to 
the executing, wherein the service controls an application that cannot be executed 
on the palm sized computer. 

2. The method of claim 1 further comprising registering the service in the 
directory of services by storing the description of the service in the directory of 
services . 

3. The method of claim 1 wherein the directory of services includes a Jini Lookup 
directory, wherein the accessing the description includes executing a Jini discovery 
protocol to locate the Jini Lookup directory and executing a Jini Lookup protocol to 
retrieve the description of the service. 

4. The method of claim 1 wherein the description of the service includes an object 
reference corresponding to an object representing the service and a set of service 
attributes including the name of the service and the physical location of the service. 

5. The method of claim 1 wherein the program code includes Java code and 
wherein the palm sized computer is executing a Java Virtual Machine to execute at least 
a portion of the program code. 
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6. The method of claim 1 wherein the program code includes code to implement a 
graphical user interface on the palm sized computer. 

7. The method of claim 1 wherein the application includes a desktop program. 

8. A method of controlling a program on a network device from a computer, the 
computer is not capable of executing the program by itself, the network device and 
computer being coupled in communications via a network, the method comprising; 
accessing a directory of services, a service in the directory of services corresponding to 

the program, the description of the service including at least a reference to program 
code for controlling the service; 
loading the program code; 

issuing control commands to the network device using the program code, the control 
commands causing the network device to control the program. 

9. The method of claim 8 wherein loading the program code includes loading the 
program code onto the computer and the issuing the control commands includes the 
computer issuing the control commands. 

10. The method of claim 8 wherein a proxy device is coupled to the network and 
wherein accessing the directory of services includes the computer accessing the proxy 
device, and the proxy device accessing the directory of services, and wherein the 
loading the program code includes loading the program code onto the proxy device, and 
wherein the issuing the control commands includes the computer issuing a set of first 
set of commands to the proxy device and the proxy device issuing the control 
commands. 

1 1 . The method of claim 10 wherein the program code includes a user interface 
program and wherein the proxy device receives the user interface program and 
generates a second user interface for the computer. 
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12. The method of claim 8 wherein loading the program code includes loading the 
program code onto the computer from the directory of services. 

13. The method of claim 8 wherein the computer includes a Palm OS compatible 
computer, wherein the program code includes Java code and wherein the directory of 
services includes a Jini directory of services. 

14. The method of claim 8 further comprising the network device registering the 
description of the service with the directory of services. 

15. The method of claim 8 wherein the program includes a desktop program. 

16. The method of claim 8 wherein the program includes a desktop program and 
wherein the services controlled include a CPU service for executing the program, a 
storage service for providing data to the CPU service and a display service for 
displaying information generated from the CPU service. 

17. A system for controlling a network service comprising: 

a network based computer service for controlling an application that cannot be executed 

by the control device; 
a directory of services including a registry of services, each service in the registry of 

services corresponding to a service on the network; 
a control device having a program for sending control commands to the network based 

computer service, the program being loaded onto the control device as a result of 

locating a reference to the network based computer service in the directory of 

services; 

a network coupled in communications with the network based computer service, the 
directory of services and the control device. 

18. The method of claim 17 wherein the control device includes a palm sized 
computer having an operating system, a network communications program, a protocol 
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program for communicating with the directory of services and wherein the program 
includes a graphical user interface. 

19. The method of claim 17 wherein the network based computer service includes a 
computer having an operating system, a network communications program, a protocol 
program for communicating with the directory of services and the application. 

20. A method for controlling a service in a network using a computer, the method 
comprising: 

accessing a description of a service, the description of the service including at least a 

reference to program code for controlling the service; 
downloading the program code; 
executing at least a portion of the program code; and 

sending control commands to the service in response to the executing, wherein the 
service includes a CPU service. 

21. A system comprising: 

means for accessing a description of a service, the description of the service including 

at least a reference to program code for controlling a service; 
means for downloading the program code; 
means for executing at least a portion of the program code; and 
means for sending control commands to the service in response to the means for 

executing, wherein the service controls an application that cannot be executed on 

the means for executing. 

22. A data processing tool for controlling an application accessible via a network, 
comprising: 

a console application including a user interface program, information about services, 
including network addresses, in a group of services accessible via the network, and a 
communication driver executing a protocol for communication of the console 
application with at least one of the services in the group; 
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an input/output device supporting the user interface program; and 
a communication port by which access to the network is available. 

23. The data processing tool of claim 22, wherein the protocol includes an exchange 
in which the console application notifies a particular service in the group of services 
which will act as an application host, of a set of services to be invoked. 

24. The data processing tool of claim 22, wherein the protocol includes an exchange 
by which the console application learns the network addresses of services in the group. 

25. The data processing tool of claim 22, wherein the protocol includes an exchange 
in which a particular service in the group of services sends the console application a set of 
user interface constructs for incorporation in the user interface program. 

26. The data processing tool of claim 22, wherein a particular service in the group of 
services comprises a slide presentation program. 

27. The data processing tool of claim 22, wherein the particular service in the group 
of services comprises an email client program. 

28. The data processing tool of claim 23, wherein the particular service ifTthe group 
of services comprises a calendar program. 

29. The data processing tool of claim 22, wherein the particular service in the group 
of services comprises a user interface program for an networked appliance. 

30. The data processing tool of claim 22, wherein the particular service in the group 
of services comprises a print service. 

31 . The data processing tool of claim 22, wherein the particular service in the group 
of services comprises a fax service. 



60 

SUBSTITUTE SHEET (RULE 26) 



WO 00/43891 



PCT/US00/01634 



5 32. The data processing tool of claim 22, wherein the particular service in the group 

of services comprises an internet browser service. 

33. The data processing tool of claim 22, wherein the particular service in the group 
of services comprises a language and/or speech translation service. 

10 

34. The data processing tool of claim 22, wherein the particular service in the group 
of services comprises a conference room reservation function. 

35. The data processing tool of claim 22, wherein the port comprises a wireless 
1 5 transmitter and receiver. 

36. The data processing tool of claim 22, wherein the port comprises an infrared 
transmitter and receiver. 

20 37. The data processing tool of claim 22, wherein the input/output device comprises a 

touch screen. 

38. The data processing tool of claim 22, wherein the input/output device comprises a 
touch screen smaller than 4 inches by 6 inches in display area. 

25 

39. A method for controlling an application executable on a particular processor 
coupled to a network using a portable computing platform, comprising: 

establishing a communication link via the network between the portable 
computing platform and the particular processor; 
30 transferring a control program to the portable computing platform via the network, 

the control program including user interface constructs for generating commands for 
control of the application; 

transmitting commands input using the control program to the particular processor 
via the communication link; 
35 transferring the commands input using the control program to the application. 
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40. The method of claim 39, wherein the application comprises a slide presentation 
application, and the commands input using the control program include commands for 
opening a presentation for display on a display coupled to the network, under control of 
the particular processor, and navigating slides within the presentation. 

41 . The method of claim 39, wherein the application comprises a slide presentation 
application, and the commands input using the control program include commands for 
editing slides within the presentation. 

42. The method of claim 39, wherein the communication link comprises a wireless 
link. 

43. The method of claim 39, wherein the communication link comprises an infrared 
link. 

44. The method of claim 39, wherein portable computing platform includes a touch 
screen, and the interface constructs include graphical user interface elements accepting 
inputs via the touch screen. 
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